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We propose a new formulation of stochastic thermodynamics for systems subjected to nonequihb- 
rium constraints (i.e. broken detailed balance at steady state) and furthermore driven by external 
time-dependent forces. A splitting of the second law occurs in this description leading to three sec- 
ond law like relations. The general results are illustrated on specific solvable models. The present 
paper uses a master equation based approach. 
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I. INTRODUCTION 

The second law of thermodynamics specifies that the 
total entropy of an isolated macroscopic system cannot 
decrease in time. This statement applies to the stages of 
the evolution in which the entropy is well defined. For 
example, for a system in equilibrium at initial and final 
times, the final entropy will be larger than the initial 
one, even though the entropy may not be well defined 
during the intermediate evolution. However, it is often 
a very good approximation to assume that the system 
is in a state of local equilibrium, so that the entropy 
is well defined at any stage of the process. For exam- 
ple, linear irreversible thermodynamics is built on such 
an assumption, allowing the use of the Gibbs relation 
to define entropy locally in terms of the slow conserved 
quantities (for example momentum, energy, and concen- 
tration of the constituents) [l|-[31- The second law can 
then be reformulated as the non-negativity of the irre- 
versible entropy production (EP) Si{t) > 0, Q: 



Sit) ^ S,{t) + S,{t) , 



(1) 



where S{t) is the entropy change of the considered sub- 
part, and Se{t) is the entropy flow to the environment. 
In some cases, the environment is idealized as being one 
or more reservoirs without internal dissipation, so that 
their entropy change is equal to minus the exchange term: 
Sr{t) = —Se{t). For a single heat reservoir, this en- 
tropy exchange is given by the energy inflow divided by 
its temperature. In the sequel, we will be mainly inter- 
ested in this situation, with the subsystem of interest, 
henceforth called the system, in contact with an envi- 
ronment consisting of one or more ideal reservoirs. The 
irreversible EP in this system is then equal to the total 
EP Stot{t) = S,it) > 0. 

In more recent developments, |6l-l32| it has been real- 
ized that one can formulate thermodynamics for small 
systems incorporating the effect of the fluctuations. 
These developments can be seen as the continuation of 
the pioneering work started by Onsager p33l - l36| . with 



as intermediate steps the fluctuation dissipation theo- 
rem [33], the theory of Gaussian stochastic processes 
and linear response [3^, [33| , and Green-Kubo relations 
[40, |4l| . The essential ingredient is to guarantee the con- 
sistency of the statistical irreversible laws on the system 
dynamics with the reversibility of the equilibrium reser- 
voirs statistics. In this new thermodynamics, also called 
stochastic thermodynamics [42, |4J], the system is de- 
scribed by a probability distribution pm evolving accord- 
ing to a Markovian master equation. The exchange of 
energy (heat) or particles with the environment and the 
other thermodynamic quantities associated to the system 
states m become stochastic variables. The rates associ- 
ated to each reservoirs satisfy the property of local de- 
tailed balance reminiscent of the fact that they always re- 
main at equilibrium. The system entropy is defined using 
the Shannon expression S — —Y^rnPm^^Pm and entropy 
balance equations of the usual form can be derived via 
the identification of a non-negative EP consistent with 
macroscopic nonequilibrium thermodynamics. A mini- 
mum EP theorem can also be proved [y, |^, \^ |44|-|46| . 

More recently, it was realized that one can study 
stochastic trajectory-dependent quantities. This is ob- 
viously the case for the energy, which is a well defined 
mechanical quantity even for single trajectories. For ex- 
ample, an explicit formulation of the first law, conserva- 
tion of total energy, was given for the stochastic trajec- 
tory of a Langevin equation by Sekimoto [S] , see also [43] . 
The stochastic exchange of energy with idealized reser- 
voirs also allows to identify the stochastic entropy flow 
into these reservoirs, and to study its statistical proper- 
ties. This led to the discovery of the celebrated fluctu- 
ation theorem: the probability distribution for the cu- 
mulated change in stochastic reservoir entropy As^ for 
a nonequilibrium steady state obeys the symmetry re- 
lation P{Asr)/P{—Asr) ~ exp(Asr) for asymptotically 
large times. We will use henceforth the notation lower 
case s for entropic contributions associated to a given 
stochastic trajectory, in contrast to ensemble average en- 
tropies denoted by a capital letter S. This result was 
first proven for thermostated systems (the main trust of 



this work being however the development of equihbrium- 
hke statistical mechanical concepts for dissipative sys- 
tems [9l-[l3|). followed by derivations for Langevin and 
master equations [isl . Il7| , and for Hamiltonian dynam- 
ics [2j, [2J]. The asymptotic nature of the result was 
linked to the large deviation properties of the character- 
istic function, and in particular to those of the currents. 
Implications include Onsager symmetry and beyond [48| 
and universal features of efficiency of thermal machines 
at maximum power |49l , |50| • Very much in the spirit of 
the fluctuation theorem, Jarzynski [ij, [ij] and Crooks 
[la . 1201 . I2I ] obtained the work theorem, see also [5ll - l53l | . 
They find that the probability distribution for the work 
w performed on a driven system, initially in canonical 
equilibrium at a specific inverse temperature /3^^, obeys 
the relation P{w)/P{—w) = exp{(3{w ~ AF)} . The over- 
bar corresponds to the probability distribution for the 
time-reversed experiment. If one assumes that, at the 
end of the driving, the system relaxes back to equilib- 
rium (at inverse temperature /3~^) then /3{w — AF), the 
so-called dissipated heat, is equal to the change in total 
entropy A.stot of system plus reservoir, so that the work 
theorem becomes a fluctuation theorem for the change in 
total entropy Astot, P{Astot)/P{-Astot) = exp(Astot) 
[23, [25|. Note that this result is valid for all times. The 
previous asymptotic fluctuation theorem can in fact be 
seen as a special case if one assumes that, aside from 
an initial transient, the steady state can be maintained 
for long enough times by appropriate driving, so that 
Astot = As -f Asr ~ Asr- The focus on the asymptotic 
form and the accent on large deviation properties is in 
our opinion a somewhat misleading representation of the 
fluctuation theorem. This form arises from the neglect of 
the system's entropy (the so-called boundary term), for 
which no readily acceptable interpretation was deemed to 
exist at the time of the first formulations of the fluctua- 
tion theorem. Furthermore, the validity of the theorem 
is typic ally compromised for a system with unbounded 
energy [54l - l58J . which is of course more the rule rather 
than the exception. We note however that current fluc- 
tuation theorems [23, [2^ do require the long time limit 
since currents are related to the Asr part of the entropy. 

A further advance consisted in the formulation of the 
equivalent of the second law at the stochastic level. This 
required the identification of a trajectory-dependent sys- 
tem entropy. Even though the idea is well known in in- 
formation theory, where — In pm is the surprise at observ- 
ing outcome m when its probability is p,„, it took some 
time before Seifert [25] identifled this quantity as the ap- 
propriate stochastic system entropy, s(t) = — lnp„(t)(i). 
Note that it depends on the actual state m{t) of the con- 
sidered trajectory at the considered time, as well as on 
the probability for this state, which itself is in general 
time-dependent. By taking this term into account, the 
asymptotic fluctuation theorem could be replaced by a 
fluctuation theorem for the total entropy, which is valid 
for all times, just as the work theorem of Jarzynski and 
Crooks. Oono and Paniconi [591 discussed an alterna- 



tive way of splitting the EP, by introducing the excess 
entropy and housekeeping heat. This led to the formu- 
lation of two other fluctuation theorems, namely one de- 
rived by Hatano and Sassa 22] (for system entropy plus 
excess entropy) and one by Speck and Seifert [6QJ (for the 
housekeeping heat). 

To close this introductory discussion, we turn to a re- 
cent development [32^, see also [2S, |30, Isll Ifi^j which 
provides a clarifying and unifying approach of the vari- 
ous fluctuation and work theorems. The total stochastic 
EP Astot is the sum of two constitutive parts, namely 
a so-called adiabatic Asa and nonadiabatic Asna contri- 
bution. Each of these contributions correspond to the 
two basic ways that a system can be brought out of equi- 
librium: by applying steady nonequilibrium constraints 
(adiabatic contribution) or by driving (nonadiabatic con- 
tribution). Note that the term "adiabatic" is used here, 
not in its meaning referring to the absence of heat ex- 
change, but in the meaning of instantaneous relaxation 
to the steady state. The crucial point is the observation 
that each of these contributions obeys a separate fluctu- 
ation relation, namely |32l ]: 



PjAstot) 
P{-Astot) 



oAsto 



P(As. 



P+{-As„a) 



PJASg) 

P+{-ASa) 



(2) 
(3) 



The superscript -I- refers to the adjoint dynamics (also 
called dual or reversal [2lj,|6l|). The aforementioned fluc- 
tuations theorems of Hatano Sassa and by Speck and 
Seifert are special case of the fluctuation theorem for 
Asna and Asa, respectively. To stress the special status 
of these theorems, we notice that they arise because of 
the two available operations to gauge the amount of time- 
symmetry breaking, namely time-reversal of the driving 
(overbar: — ) and the time-reversal of the nonequilibrium 
boundary conditions (superscript: +). Applying each of 
them separately, or both leads to the three different con- 
tributions for the EP. 

The above fluctuation theorems imply that the total, 
adiabatic and nonadiabatic entropy changes have to be 
non-negative, each taken separately: 

AStot = {Astot) > (4) 

ASna = {As„a) > , ASa ^ {ASa) > 0. (5) 

This suggests that the second law can in fact be split 
in two. There are thus three faces to the second law: 
the increase of the average total entropy, the increase 
of the average adiabatic entropy and the increase of the 
average nonadiabatic entropy. Our purpose here is to 
clarify and document further the physical properties and 
the meaning of this remarkable result. In this paper we 
will focus on the implications for a description in terms 
of a master equation. The next paper [63| deals with the 
corresponding results for Langevin and Fokker Planck 
dynamics. 



II. MASTER EQUATION 
A. Entropy balance 



We can rewrite the master equation ([6]) as 

Prnit) - E <™'(^) = E J^r^,n'it), (14) 



We first review and extend the entropy balance equa- 
tion derived previously for a Markovian process [6, 8, 29, 
|64| . Our starting point is the following master equation: 



m' 

where the rate matrix satisfies 

^ W,n,m' = 0. 



(6) 



(7) 



The transitions between states m can be due to differ- 
ent mechanism ly. Furthermore, these rates can be time- 
dependent via a control variable A. We thus have: 



W™,„. = Wrn,„^'{\t) = Y. ^n'^^m' (\ 



(8) 



A'^) 



For rates that arc "frozen" at the values W^ m'W' there 
is a corresponding stationary distribution, _p^(A), which 
we suppose to be unique (i.e. the rate matrix is irre- 
ducible) and which will always eventually be reached by 
the system. It is given by the normalized right eigenvec- 
tor of zero eigenvalue of the transition matrix: 



J^Wm.m'WKl'W^O- 



(9) 



Let us now investigate the time dependence of the sys- 
tem's Shannon entropy (Boltzmann's constant fc^ = 1): 



S{t) = -^p,„(i)lnp,„(i). 



(10) 



Using ^ and ([7]) and omitting for compactness of no- 
tation the dependence of Pm on t and of W on At we 
find: 

5(t) = -^p„,lnp^ = - Y. w^lp^,\n^ (11) 

Pm' 

rii m,7n' ,iy 

-i V iw'^'"^ V .-W^"^ V Mn — 

— 9 2_^ X^^m.m'P™'' '^ m' ,771 P-m f "^ 



= ^ E {<L'p™'-^™',™p™}in 



VF^"^ , p„ 

m,m' r'^ 
W^""} pr, 



+ 1 V WV^"^ V • -W^""^ V Tin "''-' 

T^2 2-^ 1^^771,771' Pm' m >™ ^™J jxr(l') 



m.?n' ,1^ 



w. 



It is revealing to introduce the fluxes J^\^i and corre- 
sponding forces X, 



m^m' ' 



jSm'it) = <L'(At) P,77'{t) - W^S„(AO PTuit), (12) 



w::,'{^t) PTnit) 



(13) 



where we defined 



J™,™'(i)-E4:.L'(*)- 



(15) 



The system's EP can thus be rewritten under the familiar 
form of irreversible thermodynamics 



The quantity: 



Sit)^Seit)+S,it). 



M 



(16) 



Se{t)^\ E 4rL(01n-5 



{u) ,,^,„ ^m',m(^0 

W^^'^ A\t) 
m.ra' V 'i/ 



(17) 



W''"} (At) 

m'm\^ J 



= E <L'(^*W(01n-5) 

/ W ,(Af) 

7n,7n ,v m,777'\^J 

is the entropy flow and the positive quantity 

QU\ Sr M/M (W (,\^ ^'£m'^^t)p7n'{t) 

Srit)^ 2^ W^,r,7'i>'t)P777'{t)\n—^ 



m.m' ,1/ 



w^ir'!™(A*)p™(i) 
\ E ^it'W<l'W>0' (18) 



is identified as the EP. The latter is zero if and only if 
the condition of detailed balance is satisfied: 

wi:]^,{\)Pm'^wi::l„^{\)p^. m 

We note an important property of the EP. If all the 
relevant processes v causing transitions between states 
m are not correctly identified (for example if one only 
identifies a sub-class of these processes) the EP will be 
underestimated. Indeed, using the log-sum inequality (cf. 
theorem 2.7.1 of [i^), it follows that (see also [sH-fsSJ): 



m= J2 w^;i'(A0p™'(0in 



m.m' ,u 



W^S™(At)P™(t) 



\ V^ Tl^ l\\ U\^ ^m,m'(At)pm'(0 ^n 
> 2^ VK™,m'(^t)Pm'(i)lnTTH 7^^ 7T^ >0 



Wm' ,777{^t)Pm{t) 



B. Thermodynamic interpretation 



(20) 



The above derivations and statements can be viewed as 
purely mathematical in nature and can be applied to any 
system described by a master equation. The connection 
with physics is made by associating to each mechanism 
V responsible for the transitions between system states a 
group of variables each separately at it own equilibrium 
or in other word idealized reservoirs with well defined 



thermodynamic variables (e.g. temperature or chemical 
potential). The transitions between states m due to dif- 
ferent mechanism u can for example correspond to ex- 
change of heat with different reservoirs, or the change 
in number of particles due to different chemical reac- 
tions. As a result the transition rates associated to a 
given mechanism v need to obey the condition of local 
detailed balance 

Wl^la' i^t)pZ {\UV) = Wl%S\t)p';?A\ul^): (21) 

where the equilibrium distribution p'^'^{Xt,v) is the sta- 
tionary distribution that would be reached by the sys- 
tem if only a single mechanism v were present and for 
the frozen value of the control variable A = Af. In case 
multiple mechanism are present, each reservoir tries un- 
successfully to impose its equilibrium distribution on the 
system resulting in a stationary distribution that does 
not satisfy the detailed balance condition (ITO)) except if 
their thermodynamical properties are identical, making 
the distinction between the various mechanism useless. 
As we have seen in (I^Ul) , a incorrect identification of the 
various reservoirs would underestimate the EP and could 
lead to believe that a system is at equilibrium while it it 
not [8] . The assumption that the same expressions for the 
transition probabilities in (I2ip can be used when the con- 
trol parameter becomes time-dependent is based on an 
assumption that the idealized reservoirs relax infinitely 
fast to their equilibrium compared to the timescales of 
the system dynamics. 

By an argument of physical consistency, it follows that 
Si{t) is also equal to the total EP Stot{t) in system plus 
environment. Indeed, since one implicitly assumes that 
the environment remains at equilibrium at all times us- 
ing (mj, it docs not have an internal EP of its own. 
Otherwise, the above description in terms of the sys- 
tem alone is not complete, and one needs to include the 
description of the irreversible processes taking place in 
the environment. For the same reason, the entropy flow 
is equal to minus the entropy increase in the reservoir 
Se{t) = —Sr{t). The microscopic origin of these rela- 
tions has been recently clarified [66|. 

In order to make the thermodynamic interpretation of 
the stochastic dynamics transparent, we now explicitly 
evaluate the various entropies for multiple heat reser- 
voirs satisfying local detailed balance with respect to the 
canonical equilibrium distribution at the inverse temper- 
ature of the corresponding reservoir, i.e. (PT|) becomes: 



W. 



(i') 



(A) 



W^-V (A) 

ra' ,rn\ / 



= exp{-/3M(e„,(A) -e™,(A))}, (22) 



where em(A) is the energy of the system when in the 
state m for the value A of the control variable. We note 
that in order to make the connection between the system 
Shannon entropy (llOp and the true thermodynamic en- 
tropy, one might have to add a contribution ^^ SmPm{t) 
to the entropy, where 5*^, is the entropy of each level 
m |3ll. 1671. 1681]. However, for simplicity, we now assume 



that the index m refers to the non-degenerate microscopic 
state of the system (i.e., Sm = 0). It immediately follows 
from ((22|) that the entropy flow (fT7| takes on the familiar 
thermodynamic form of heat flux over temperature: 

^e(0=E/^^''^^^''^W' (23) 

where the heat from the i^ reservoir is given by 

Q('^)(i)= J2 .tl,it)e^'i\). (24) 

m.m' 

Since the system energy is obviously given by: 

S(t)==^e,„(A)p™(t), (25) 

m 

we find from the first principle of thermodynamics (en- 
ergy conservation). 



that the work is given by: 



(26) 



(27) 



This illustrates that the local detailed balance conditions 
with the reservoirs (P^ leads to a proper formulation 
of he first (pS)) and second pB|) principle of thermody- 
namics. We note that it is possible to include particles 
exchanges with the reservoirs [501 • In the latter case, 
work can be nonzero even in absence of driving. 

We should finally mention that the system could also 
be driven out of equilibrium by a nonconservative force 
instead of different reservoirs. In this case, the local de- 
tailed balance condition (|21l) is not satisfied and even in 
presence of a single reservoir, the detailed balance condi- 
tion p^ will not be satisfied at steady state. 



C. Adiabatic and nonadiabatic entropy balance 



We next note that the force X p3l) can be split in an 
adiabatic contribution A and a nonadiabatic contribution 

<l' W = <l'(A*) + iV„,„,(t) (28) 



<U'(A*) = ln 



Nm,7n'{t) =ln 



W^™'!™(A*)p,^^(At) 

Pt^{Xt)p,n'{t) 
P'Z{M)Pra{ty 



(29) 



(30) 



The total EP can thus also be split into an adiabatic 
and nonadiabatic contribution: 



S,{t) = Stotit) ^ Sa{t) + S^a{t) 



(31) 



with: 



Sa{t)^\ E Jil-W<L(A0>0. 



(32) 



7n,m' M 



^na(t) - 2 ^ J-,n,m'{t) N„,,„At) > 0. (33) 

7n^m' 

Pm{t) 



y^^Prait)\n 



P^i(At) 



We make a number of remarks. First and most impor- 
tantly, both Sa{t) and Sna{t) are non- negative EPs. This 
follows from Jensen's inequality, — In a; > 1 — x for x > 0, 
together with ([7]) and ©■ The non- negativity of these 
quantities is in agreement with the fact that the tra- 
jectory entropies Sa and Sna obey detailed fluctuation 
theorems |3^]. Second, it is clear that the nonadiabatic 
thermodynamic force, and hence the nonadiabatic EP, is 
zero in the adiabatic limit Pm{t) —>■ pf^{Xt). This will for 
example be the case when the relaxation to the steady 
state is extremely fast, and in particular faster than the 
time-scale of the driving At. This observation justifies a 
posteriori the name given to each contribution. Third, 
we note that one does not need to identify the sepa- 
rate mechanisms i' by which the transition between states 
takes place for the evaluation of nonadiabatic EP, or its 
corresponding fluxes and forces. It is function only of 
the coarse grained transition probabilities W = J2i, W ■ 
Finally, we note for a system subjected to a nonconser- 
vative force and in contact with a single reservoir, the 
adiabatic EP is the Housekee ping h eat (divided by the 
temperature of the reservoir) [22, [5^, [62, Ifi^l ■ 

The fact that there are two contributions to the total 
entropy production, which are separately non-negative, 
suggests that the second law can be "split in two". An 
elegant way to do so is by the introduction of the so-called 
excess entropy change [22, 113 : 



^ex\t) — „ 2_^ '^" 






(34) 



^Pm{t)\np'^{Xt 



One easily verifies that: 



S{t) = -SeAt)+Sna{t) 



(35) 
(36) 



Written under this form, we see that the changes in sys- 
tem and reservoir entropy both have a structure similar 
to the original second law (TTBl) : they consist of the sum of 
a reversible and an irreversible term. By summing these 
two relations we recover (I5T]) . The relations (pij) . ([55)) 
and ([5S|) thus represent the three faces of the second law. 



III. SPECIFIC CLASS OF TRANSFORMATIONS 

Additional comments can be made when considering 
the specific classes of transformation discussed below. To 
proceed, it is useful to split the nonadiabatic entropy ([55)) 
into a "boundary" and a "driving" part (23]: 



where 



S„a(t)=St{t)+Sd{t), 

,,,..^(j:,„,„„^) 



(37) 

(38) 
(39) 



The boundary contribution only depends on the initial 
and final distribution of the considered transformation 
while the driving part is only non-zero when the external 
perturbation evolves in time. 



A. Transient relaxation to steady state 

The system is supposed to be in an arbitrary state 
when the external driving is switched off, say at i = 0. 
For i > 0, we have that At = A is time-independent, 
implying Sd{t) — 0, and therefore: 



Sna{t) = Sb{t) = -H{t). 



(40) 



Here, H{t) is the relative entropy (or Kullback-Leibler 
entropy) [65| between the actual and the steady state 
distribution: 



H{t) ^ D{p{t)\\p^\\)) 



n„i(i)ln 



Pr,i{t) 

P,f.(A) 



> 0.(41) 



We conclude that H{t) is a Lyapunov function, decreas- 
ing monotonically in time until the probability distribu- 
tion reaches its steady state value. This proof of conver- 
gence to the steady state is well know [ZOl, but its con- 
nection to the nonadiabatic EP has never been pointed 
out. The above result can also be viewed as a gener- 
alization of the minimum EP principle (MEP) [8|: the 
nonadiabatic entropy assumes its minimum value equal 
to zero at the steady state, but the latter need not be 
close to equilibrium. 



B. Transitions between steady states 

We consider a system that is initially in a steady state 
Pm(0) = p^(At;). Then somewhere between ti and i/ the 
time dependent force changes and after an asymptotically 
long time T the system has reached its new steady state 
distribution Pm{T) = p**(Atj). Let us consider the to- 
tal nonadiabatic entropy change during the time interval 
T. The boundary term is zero and the driving term is 



nonzero between ti and tf. The relation ((35)) becomes 
the second law of steady state thermodynamics [22, [53] : 



ASnaiT) - AS4T) - ASeAT) + AS'\T) > 0,(42) 

where AS"**(T) is the system entropy change between the 
initial and final steady state. 



C. Time dependent cycles 

We consider a system that is subjected to a time- 
periodic perturbation with period T. After an initial 
transient, the probability distribution and the various 
EP's will also become time-periodic functions with the 
same period. Because of continuity, we have p,„(0) = 
p„i{T). Furthermore Pm{^o) = Vmi^T)- Hence both the 
boundary term along a cycle as well as the system entropy 
change over a period are zero, AS{T) = ASh{T) = 0, and 
the relation (1551) becomes 

ASna{T) = ASeAT) = ASd{T) > 0. (43) 

We also have, using (fT6|) . that: 

AStotiT) = ASriT) = ASe4T) + ASa{T) > (44) 

D. Perturbing the steady state 



We now turn to the nonadiabatic EP which using ([33 
and (1451) now reads: 



Sna{t) = - ^ 5pm{t) lu 



Pm{t) 



>0. 



(52) 



In analogy to the total EP [which can be written in a bi- 
linear form in term of flux and forces (|18|) ]. we can write 
the nonadiabatic EP as a bilinear form of the nonadia- 
batic flux and force as: 



i^na{t) 



noting however that: 



N„ 



,(t) 



E 



5p-n 



SJ„ 



>0, 



OPm , ^/r2 \ 

-zir + ois p). 



(53) 



(54) 



We conclude that the adiabatic EP has a constant (zero 
order) and first order terms in the deviation dp around 
the stationary state p^, while the nonadiabatic EP is of 
second order. In general, since SSa(t) can be negative, we 
can find situations where Sa{t) and even Si{t) are smaller 
then Sa{X). This is the well known result that the MEP 
for the total EP does not hold around nonequilibrium 
steady states but does around equilibrium steady states 
since <)„,(A)= oil. 



Finally, we investigate how the adiabatic and nonadi- 
abatic EP change upon applying a perturbation around 
the steady state distribution: 

p^it)^pti\)+SpUt). (45) 

Clearly, the flux (|12|) can be split as: 

^il'(i) = ^il'(A) + Sj!:l,{t), (46) 

where 



(47) 



r(^) 



A>^) 



{^) 



5JZU'{t) = W^r™'(A) Spra'it) - WZ\^{\) Sp„,{t) 

(48) 

The adiabatic contribution to the EP can be written 
as the sum of two contributions 



Sait)^SaiX)+SSa{t)>0, 



(49) 



with: 



^<^(A) = ^ E '^iL'(A)<U'(A)>0 (50) 

m,m',i/ 

ssa{t) = l Yl '5J^Lw^™L'(A)- (51) 



IV. APPLICATIONS 

A. Two level system 

We consider a system with two levels, m = 1,2, with 
Pm the probability to find the system in level m. Due to 
conservation of total probability, the master equation ([6|) 
can be reduced to a single differential equation for the 
probability p = p2 = 1 — Pi to be in the level 2: 



m^-i{^t)p{t) + w2i{Xt), 



(55) 



with 7(A) = W2i{X) + Wi2{X). The steady state solution 
isp^*(A) = iy2i(A)/7(A). 

We first present the general solution for a periodic per- 
turbation, with a period T, without specifying the form 
of the rates at this stage. We focus on the long-time 
regime, where all transients have disappeared, and the 
time behavior oi p{t) itself is periodic with period T. We 
need to solve ([55]) subject to periodic boundary condi- 
tions p(0) = p{T). For simplicity, we focus on the case of 
a piece-wise constant perturbation. Hence, the driving 
period consists of two regimes / and //, namely At = A/ 
for < t < i/, and At = A// for tj <t <T. The solution 



of 



reads [pf = p'\Xi), pfj = p'^Xn)): 



0<t<ti: 
ti <t <T 



p{t) = {p{T)~pf)e~'"'+pf (56) 

pit) = {p{tj) -pfj)e-''"^'-*'^ +pfj. (57) 



Using the matching condition (continuity of p) at the 
transitions between regions / and //, we find that: 



(flit I 



-lii{T-ti) 






(1581) and (|59l) with ^ and ^ provide the exact and 
exphcit long-time solution of (j55p under piece-wise con- 
stant periodic driving. 
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FIG. 1. (Color online) (A) Actual probability distribution 
along the cycle and stationary solution corresponding to the 
instantaneous values of the driving displayed in the inset. 
(B) The total, adiabatic and nonadiabatic entropy production 
along the cycle. The Bose rates (|64|l are used with F = 0.5. 
Also r = 1 and i/ = 0.4. 

We now turn to the entropies. We consider the case 
of two different reservoirs v = L^R, with corresponding 
transition rates W = Y,u W-^^ The entropies ^, ^ 
and (1551) become 



^to* = EK2^p-^2r^(i-p))i^^ 



wi',^p 



w^';\i-p) 

Sa = Y. (W^i2^P - Wt,\l p)) In -^^(^ 



W^i'^V^* 



w^2V(i-p^*) 



Sua = p{t) In 



(i~pK 

(1 — p^*)p 



(60) 



(61) 



(62) 



As an illustration, we consider two physical models 
that are described by the above two-level master equa- 



tion. The first is a single electron quantum level in con- 
tact with a left (L) or right (R) electronic reservoirs 
{h' ~ L,R). The level can thus be empty, 1 — p, or filled, 
p, and transitions between these two states correspond 
to the exchange of an electron with one of the two reser- 
voirs. The rates are given by the Fermi golden rule rates 
for each of the reservoirs [TJJ 



w^-;^ - r/(A,) 



w^ir^ 



r(l-/(A,)), (63) 



where f{x) = (exp {cc} -I- 1)^^. The thermodynamics of 
this model has been discussed in Refs. [22)[Z3l- The time 
dependence of the rates can result from either the exter- 
nal control of the energy of the level (e.g. with an electric 
field) or from the control of the reservoirs chemical po- 
tentials. 

Our second model represents two level atom interact- 
ing with a left (L) and right (R) reservoir of thermal 
light. The rates describing the transitions between the 
levels are then given by: 



ri-^) 



w^2i^ = r7i(A,) 



T4^ir' = r(i 



i{X,)), (64) 



where n(x) — (exp{a;} — 1)""'^. The time dependence of 
the rates can either come from an external control of the 
energy spacing between the levels or, which is less realis- 
tic, from the control of the temperature of the reservoirs. 
The probability distribution as well as the various en- 
tropies around the cycle can be analytically evaluated. 
For the purpose of illustration, we reproduce the typical 
behavior in Figs. ^ and ^. 



B. Chemical reaction model 



Consider the following chemical reaction: 



A(^)^ 



M ^ 



aw 



(65) 



Denoting by m the number of particles of species AI and 
using 1/ = L, i?, we find that the corresponding transition 
rates are given by 

W^l.^^^ki^^m , wi%„ = 4'')AM(t). (66) 

The concentrations of A^^^ (t) and A'--^'> (t) are externally 
controlled in a time dependent manner. Rescaling time 
by fcL -|- fc], , , we can redefine the rates as 



" m—l.m " ' 

where we introducted 



' m-\-l,m 



>M 



vM 



M 

^M 



iH 






+ k 



(R) 

M 



dL) , ,iR) ■ 



(67) 



(68) 




FIG. 2. (Color online) Same as Fig. [Tl but using the Fermi 
rates (|63[) instead of the Bose rates. 



The resulting maste r eq uation reads (m = 0,1,... with 
convention p_i = 0) 70]: 



Pn 



^{a('')(m + l)p„+i + A. 



t Pm^l 



-(aI'') +«('') m)p,„} (69) 

= (m + l)pm+l + hPm-1 - (At + m)pm. 

From first to second line, we used ^^^ A^ — At and 

X^i/Q^^"'' — 1- ^1- (Ell) can be most easily solved by 
switching to the generating function: 



E s>™(i)- 



(70) 



F{s,t)^ 

Using dnSI) and ([701), one finds: 

dtF{s,t) = (s - l)AtF(s,i) -{s- l)dsFis,t). (71) 

For purpose of illustration, we focus on a specific time- 
dependent solution of this equation, for which the explicit 
analytic expression can be obtained. Indeed, one readily 
verifies by inspection that: 



Fis,t) 



- pr-n{t){s-l) 



(72) 



is an exact solution of (j7ip . propagating in time. As 
suggested by the notation, rh(t) is the time dependent 
average: 



oo 
TO = fh{t) — 2, ITT'Pmit) 
?n— 



(73) 



which obeys the following equation: 

fh{t) — Xt — TO(i). 
Its exact time-dependent solution is given by: 



(74) 



i{t) 



- p-(«-to)i 



{m{to) + / dre("-*°)A,}. (75) 



to 



We conclude that the propagating Poissonian distribu- 
tion: 



, , TO'" 

Pm(t) = — j-e 
to! 



(76) 



with the average given by (j75p . is an exact time- 
dependent solution of the master equation. The corre- 
sponding frozen steady states are 



p™W 



with 



ml 



= Xt 



We note, using (1571) and d75|) . that 



W 



H 



771 + 1,771- 



Pmit) 



,H 



'^''m,m+lPm+l\'') ^ I 



(77) 



(78) 



(79) 



We see that at steady state, the detailed balance condi- 
tion is not in general satisfied. Equilibrium is only at- 
tained if A( = a^'^-'At. We thus have two mechanism 
bringing the system out of equilibrium, the breaking of 
detailed balance by the steady state and the time depen- 
dent driving from A^ . 

Using these analytical results, we can now obtain ex- 
plicit results for the entropies ([SI]) . (|32|) and (|33|) : 



Stot{t) 
Sa{t) 



E' 



a^'^'m, ■ 



Ai^))ln" 



M, 



i'^) 






{a 



M, 



v('^)^ 



In- 



t-('')to^* 



A 



-TTiln ■ 



At 



(to - At) In 



t 

TO 

aT' 



(80) 

(81) 
(82) 



which are all positive. Not surprisingly, the nonadiabatic 
contribution vanishes in the adiabatic limit to = At. On 
the other hand, the adiabatic contribution vanishes un- 
der the instantaneous detailed balance condition Aj — 
a^'^-'At (remembering that At = to"*). 

To compare the results with the two-level model, we 
again assume that the system is subjected to a periodic 
piece-wise constant driving of period T. One finds: 



0<t<ti: 
ti <t<T 



m{t) : 
fh{t) 



*(TO(r)-HA/(e*-l)) (83) 

-(*-*«)(TO(i,)+A,Ke*-*^-l)), 




The resulting behavior of the probabiUty distribution and 
the various EP around the cycle are shown in Fig. (|3]). 



FIG. 3. (Color online) (A) Actual average m(f) and station- 
ary average At along the cycle [they uniquely determine the 
actual and stationary probability distribution (|76|) and (|77|) ] 
corresponding to the instantaneous values of the driving dis- 
played in the inset. (B) The total, adiabatic and nonadiabatic 
entropy production along the cycle. Also T — 5 and ti = 2. 



V. CONCLUSIONS 



In this paper, we started by showing how to identify 
entropy and entropy production for a stochastic dynam- 
ics described by a Markovian master equations with time 
dependent rates. The key new element is that the entropy 
production can be "split" in two parts, each satisfying a 
second law like relation. By assuming that the rates sat- 
isfy a local detailed balance condition, we have also shown 
that the dynamics provides a nonequilibrium thermody- 
namics description of the system. The thermodynamic 
implications of this new splitting remain to be properly 
understood. To progress in this direction we calculated 
the various entropies on a different exactly solvable mod- 
els. In the companion paper |63j . we proceed similarly 
for a stochastic dynamics described by a Fokker-Plank 
equation. 



where 



m(ti) 



m{T) 



A/(c-'^-l) + A//(e- 



-") 



A/(c*^-l)-f A//(e^-e*0 



(84) 
(85) 
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